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Abstract 

\ We give the first example of a non-linear residually finite 1-related group: (a, t | a* = a 2 }. 

i— i 

1 Non- linear groups 

o 

£ ■ HNN^(G) = (G,t | t^t" 1 = g e G) 

is called an ascending HNN extension of G (or the mapping torus of the endomorphism 0). In 
particular, the ascending HNN extensions of free groups of finite rank are simply the groups 
given by presentations (x\, x n , t \ txit~ x = Wi,i = 1, ...,ra), where w\,...,w n are words in 
xi, ...,x n generating a free subgroup of rank n. 

In BS , Borisov and Sapir proved that all ascending HNN extensions of linear groups are 
residually finite. After |BSj . the question of linearity of these groups became very interesting. 
■ This question is especially interesting for ascending HNN extensions of free groups because most 

of these groups are hyperbolic I.Kap , and because calculations show that at least 99.6% of all 
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Let <p be an injective endomorphism of a group G. Then the HNN extension 



1-related groups are ascending HNN extensions of free groups BS . 

Let H = HNN^(F n ) be an ascending HNN extension of a free group. If n = 1 then H is a 
Baumslag-Solitar group BS(m, 1), so it is inside SL2(Q). If n = 2 and <p is an automorphism 
then the linearity of H follows from the linearity of Aut(F2). The linearity of Aut(i<2) follows 
from two facts: Dyer, Formanek and Grossman DFG reduced the linearity of Aut(i<2) to the 
linearity of the braid group B4; the linearity of B4 was proved by Krammer |Kraj . 

It is known that in the case when <p is not an automorphism, the situation is different. 

5_i ' 

Proposition 1 (Wehrfritz, [WJ, Corollary 2.4). The group (a,b,t \ tat' 1 = a k ,tbt~ 1 = b l ), 
with k, I £ {1, —1}, is not linear. 

The proof 1 in [W] uses the action of SL n (K) on the Lie algebra sl n (K) to deduce that if a, b, t 
are matrices satisfying the relations of the group then some powers of a and b generate a nilpotent 
subgroup. One can also prove this statement by using the fact that if o, t are matrices with 
complex entries such that tat -1 = a k , \k\ > 1, then a k ' 1 £ [/(i -1 ) = {x \ hin^^oo t~ n xt n = 1} 
and the well known fact that U(t~ l ) is a nilpotent group for every matrix t. 

The following lemma is useful when dealing with ascending HNN extensions of groups. 

*The first author is grateful to the CNRS of France for granting her the delegation CNRS status in the 
2003-2004 academic year. The research of the second author was supported in part by the NSF grant DMS 
0072307. 

x In the first version of this paper that appeared in the arXiv, we gave a complete proof of this statement 
because we were unaware of |Wj . We are grateful to Professor Raptis for providing this reference. 
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Lemma 2. Let <fi be an injective endomorphism of a group G. Suppose that (j) k is not an inner 
automorphism of G for any k ^ 0. Then a homomorphism 7 of H = HNN^(G) is injective if 
and only if the restriction 0/7 on G is injective. 



Proof. This follows immediately from the fact [BS that every element of H is of the form 
t~ p wt q , where p, q > 0, w G G. Indeed suppose that 7 is injective on G but j(t~ p wt q ) = 1 
where t~ p wt q ^ 1. Then "f(wt q ~ p ) = 1, and so q 7^ p. Hence j(w) = "f(t p ~ q ). Let m = p — q. 
We can assume that m > 0, otherwise replace w by w' 1 . This implies that for every u £ G, 
r )(t m ut~ m ) = ^{wuw' 1 ). The injectivity of the restriction of 7 on G then implies that t m ut~ m = 
wuw' 1 . Since t m ut~ m = <f) m {u), we get 4> m (u) = wuw' 1 . Hence 4> m is the inner automorphism 
induced by w, a contradiction. □ 

Corollary 3. The group H = (a,b,t \ tat~ l = a k ,tbt~ l = b l ) is linear if and only if k,l € 
{1-1}- ' 

Proof. If k, I £ {1, —1} then <ft is an automorphism and H is linear, say, by the results of DFG 
and |Kraj cited above (one can also use the fact that the group has a subgroup of finite index 
isomorphic to F2 x Z). If both k, I {1,-1}, we can apply Proposition It remains to consider 
the case k {1,-1}, I £ {1,-1}. Then it is easy to see by Lemma |21 that the subgroup 
(a, bab^ 1 , t 2 ) is isomorphic to the group (x, y, t \ txt~ l = x k , tyt^ 1 = y k }, and so it is not linear 
by Proposition n Hence the group H is not linear as well. □ 

Not much is known about the linearity of 1-related groups. Note only that all residually 
finite Baumslag-Solitar groups (i.e. HNN extensions of cyclic groups) |Mej are linear |Voj . 

The following theorem gives the first example of a non-linear residually finite 1-related group. 

Theorem 4. The group (a, t \ t 2 at~ 2 = a 2 ) is residually finite but not linear. 

Proof. Using Magnus rewriting procedure, this group can be represented as an HNN extension 
(a,b,t I tat' 1 = b,tbt~ l = a 2 ), so it is residually finite by |BSj . The subgroup of that group 
generated by {a,b,t 2 } is isomorphic (by LemmaEJ to (a,b,t \ tat' 1 = a 2 = b 2 ) which is 
not linear by Proposition ^ The isomorphism takes a to a, b to b, t to t 2 . □ 

Problem 5. Is it true that HNN ( ^ ) (i ? n ) is always linear if cf> is an automorphism? 

Problem 6. Are there hyperbolic non-linear ascending HNN extensions of free groups? In 
particular, is the group (a, b, t \ tat' 1 = ab, tbt' 1 = ba) linear (the fact that this group is 
hyperbolic follows from |I.Kap| )? 

The group (a,b,t \ tat' 1 = ab^bt' 1 = ba) is actually a 1-related group (a,t j [[a,t],t] = a). 
The fact that this group does not have a faithful 2-dimensional representation follows from 
|FLB Jj . Moreover, results of |FLR,| (and prior results of Magnus jM]) imply that most 1-related 
groups do not have faithful 2-dimensional representations. 

We conjecture that the answer to Problem El is that most groups HNN^Fk), k > 2, are non- 
linear provided <f> is not an automorphism. In particular there are many non-linear hyperbolic 
groups among these HNN extensions. 

Note that there are examples of non-linear hyperbolic groups yet, although M.Kapovich 



M.Kap has an example of a hyperbolic group which does not have faithful real linear represen- 



tations. 
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2 Added 10/12/04: It is easy to show using Kapovich's argument that the group does not have faithful repre- 
sentations over any field. 
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Proposition ^ and Corollary |3] give examples of non-linear ascending HNN extensions of 
linear groups. Non-ascending HNN extensions with this property are much easier to find: some 
of these HNN extensions are not even residually finite (say, the Baumslag-Solitar groups (a,t \ 
taPt^ 1 = a 3 )). Residually finite non-linear HNN extensions of linear groups were constructed in 
particular by Formanek and Procesi [FPj . They proved that the HNN extension of the direct 
product Fk x Fk , where one of the associated subgroups is the diagonal and the other one is one 
of the factors, is residually finite but not linear. This was the main ingredient in the proof in 
|FPj of the non-linearity of Aut(i ? n ), n > 3. 

2 Representations in SL2(C) 

By Lemma 1^1 finding a copy of H = HNN^i 7 ^) in SL n (K) amounts to finding a /c-tuple of 
matrices (A\, A^) that freely generate a free subgroup, and which is a conjugate of the 
/c-tuple (cj)(Ai), </>(Afc)). In the case when k = 2, n = 2, K = C one can use the fact 
that conjugacy of two pairs of 2 by 2 matrices (A,B), (C,D) implies the system of equalities 
trace(A) = trace(C), trace(-B) = trace(D), tiace(AB) = trace(CD). The converse statement is 
"almost" true: one needs to exclude the case when 

traceQA, B]) = trace(^) 2 + trace(5) 2 + trace(AB) 2 - trace(,4)trace(.B)trace(AB) -2 = 2 (1) 

(in that case A, B generate a solvable group |Bowj ) . Using the fact that for every word 
u = u(A,B) in matrices A,B G SL2(C), trace(u) can be expressed as a polynomial in trace(^4), 
trace(-B), tiace(AB), we get a system of three equations with three unknowns. The correspond- 
ing algebraic variety will be called the trace variety of the group HNN^(Ffc). 

In most cases that we considered, the trace variety was O-dimensional. But the next example 
shows that the trace variety may have dimension > 1 and the group still may not have a faithful 
2-dimensional representation. 

Example 7. The group H = (a,b, i|tat _1 = a^bt^ 1 = [a,b]} does not have a faithful 2- 
dimensional representation. The trace variety of this group is a union of two curves, but it 
consists of non-faithful representations. 

Consider any representation of H in SL2(C). So we assume that a, b are 2 by 2 matrices with 
determinant 1. Let us denote trace (a) = x, trace (6) = y, trace (ab) = z. It is easy to see using 

trace(5^ 2 C) = trace(A)trace(5^C) - trace(#C) (2) 

(this is essentially the Cay ley-Hamilton theorem for matrices in SL2) that we have the following 
system of equations: 

x = x 

y = trace(a6a _1 6 _1 ) = —2 + x 2 + y 2 + z 2 — xyz 
z = trace(a[a, b]) = by J2J x ' trace(a6a _1 6 _1 ) — trace(6a _1 6 _1 ) 
= x ■ trace(o6a _1 6 _1 ) — x = xy — x 

Plugging z = xy — x into the second equation, and solving for y, we get y = 2, x = z or 
y = x 2 — 1, z = x 3 — 2x. Thus the trace variety is a union of two curves. If y = 2, x = z then 
(a, b) is solvable by (^Q), so the representation is not faithful. Now let y = x 2 — 1, z = x 3 — 2x. 
Consider the word w = ab~ 1 a~ 1 ba~ 1 b~ 1 a. It is not difficult to compute the corresponding trace 
polynomial: 

trace(w) = — 3y — 4xz + 5yx + xz — 2yx z + yz + y — y x +yxz + xz — yx. 
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If we plug in y = x 2 — 1,2 = x 5 — 2x into this polynomial, we get 2. Similarly, the trace 
polynomial of the word wa is 



x A y 2 z — x 5 y — x 3 y 3 — 2x 3 yz 2 +x 4 z — x 2 y 2 z+x 2 z 3 + 6x 3 y+2xy 3 + 3xyz 2 — 5x 2 z — y 2 z — z 3 — 7xy+3z. 

If we plug in y = x 2 — 1, z = x 3 — 2x, we get x. Hence trace(u>) = 2,trace(w;a) = trace(a) = x, 
whence trace ([w, a]) = 2 (see Q) and u> and a generate a solvable subgroup. Therefore for 
every value of x the corresponding representation of the group H is not faithful. (In fact it is 
not difficult to show that the relation (a 2 6) 3 = 1 also holds, so in this case (a, b) has torsion.) 

Similarly the trace variety of the group (a,b,t \ tat -1 = a,tbt~ l = {ba)b{ba)~ l ) is two- 
dimensional, but this group does not have faithful representations in SL/2(C). 

In fact we do not know the answer to the following question. 

Problem 8. Are there any ascending HNN extensions of Fk, k > 1, which have faithful 2- 
dimensional complex representations? In particular, are there free non-cyclic subgroups in 
SL2(C) which are conjugate inside SL2(C) to their proper subgroups? 



References 

[BS] A. Borisov, M. Sapir. Polynomial maps over finite fields and residual finiteness of 



mapping tori of group endomorphisms, arXiv:math.GR/0309121 



[Bow] B.H. Bowditch. Markoff triples and quasi-Fuchsian groups. Proc. London Math. Soc. 
(3) 77 (1998), no. 3, 697-736. 

[DFG] J. L. Dyer, E. Formanek, E. K. Grossman. On the linearity of automorphism groups 
of free groups. Arch. Math. (Basel) 38 (1982), no. 5, 404-409. 

[FLR] B. Fine, F. Levin, G. Rosenberger. Faithful complex representations of one relator 
groups. New Zealand J. Math. 26 (1997), no. 1, 45-52. 

[FP] E. Formanek, C. Procesi. The automorphism group of a free group is not linear. J. 
Algebra 149 (1992), no. 2, 494-499. 

[I.Kap] I. Kapovich. Mapping tori of endomorphisms of free groups. Comm. Algebra 28 (2000), 
no. 6, 2895-2917. 

[M.Kap] M. Kapovich, Representations of polygons of finite groups, preprint, 
http:/ /www. math. ucdavis.edu/~kapovich/EPR/gr4.ps . 

[Kra] D. Krammer. The braid group B4 is linear. Invent. Math. 142 (2000), no. 3, 451-486. 

[M] W. Magnus. Two generator subgroups of PSL (2,C). Nachr. Akad. Wiss. Gottingen 

Math.-Phys. Kl. II 1975, no. 7, 81-94. 

[Me] S. Meskin. Nonresidually finite one-relator groups. Trans. Amer. Math. Soc. 164 
(1972), 105-114. 

[Vo] R.T. Vol'vachev. Linear representation of certain groups with one relation. Vestsi 

Akad. Navuk BSSR Ser. Fi z.-Mat. Navuk 1985, no. 6, 3-11, 124. 



4 



[W] B.A.F. Wehrfritz. Generalized free 

Soc. (3) 27 (1973), 402-424. 

Cornelia Dru^u 
Department of Mathematics 
University of Lille 1 
and UMR CNRS 8524 
Cornclia.Drutu@math.univ-lillel.fr 



products of linear groups. Proc. London Math. 

Mark V. Sapir 
Department of Mathematics 
Vanderbilt University 
msapir@math.vanderbilt.edu 



5 



